System resource management schemes play an important role in determining the performance and latency characteristics of a data processing system. Most data processing engines use either an on-demand policy or a static mapping policy as their backend resource management infrastructure. The on-demand resource management policy, often referred to as a thread-pool, is used in many data-driven and compute-intensive engines such as Hadoop and Ceph. The static resource management is a basis of most commercial, high performance operating systems (OSes) for data storage systems or data processing engines such as SCYLLA.
The on-demand policy is good at utilizing compute resources (e.g., a central process unit (CPU)) based on a fair and time-sharing scheduler of an operating system when executing processes or threads. However, the on-demand policy often can create a high CPU contention between input/output (I/O) and compute tasks resulting in a longer latency. The static mapping policy can alleviate the CPU contention by assigning I/O and compute tasks to different physical cores of the CPU. While the static mapping policy can enjoy a shorter latency, the static mapping policy would not be ideal when the I/O workload is unpredictable, or there are not sufficient requests for the pre-assigned threads and compute resources. Insufficient requests to a pre-assigned task can cause the compute resources to idle, leading to a sub-optimal performance.